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ABSTRACT 



Cost-effectiveness analys is and cost-benefit analysis 
are two related yet distinct methods to help decision makers choose 
the best course of action from among competing alternatives. For both 
types of analysis, costs are computed similarly. Costs may be reduced 
to present value amounts for multi-year programs, and parameters may 
be altered to show differing cost results. Effectiveness is typically 
an indirect constructed measure of selected program outcomes. For the 
same unit cost, one can determine which option yields the most 
effectiveness, or which program produces a given level at the lowest 
cost. Benefits are program outcomes expressed as money. Benefits are 
not necessarily recorded by the accounting system of the institution 
sponsoring the program. For either types of analysis, the rules for 
evaluating and deciding are essentially the same. The first step in 
any cost analysis is to determine the cost-center of analysis (the 
agency, department, programs, sets of programs, or program elements). 
The constant yprdstick in effectiveness analysis is the desired 
outcome, but in benefits analysis, the constant yardstick is monetary 
value. Neither method is a panacea, but both are helpful in decision 
making. Six figures are included. Appendix 1 presents a cost 
categorization format. (SLD) 
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COST-EFFECTIVENESS & COST BENEFIT ANALYSES: 
CONFRONTING THE PROBLEM OF CHOICE 



Bob Johnson, Executive Vice President and Divisional Manager of 
Widgets Manufacturing Corporation, is concerned about the sluggish 
sales and rising costs of his division. The key issue facing him 
now is what to do. Among the proposals he is considering are 
these : 

* Put into place a revised sales incentive system for his 
sale representatives. Projected marginal cost: $25,000. 

* Train his supervisors and their employees in certain work- 
plannino and control techniques, along witn a brief orien- 
tation to the division's current business situation. 
Projected cost: $20,000. 

* Improve worker motivation through a "job enrichment*' 
program of increased responsibility and worker control of 
their work. Projected cost: $35,000. 

* Install a new set of machinery at certain points in the 
work-flow to replace deteriorating stock. Projected cost: 
$150,000. 

* Begin a "quality circles" program of worker problem-solving 
of production problems. Projected cost: $20,000. 

And, Bob figures, his other option is always to do nothing and 
hope that either the economy in general picks up or that some of 
these expense problems dissolve from tighter controls/ Coming up 
with $25,000 to $35,000 from existing budget manuevers could be 
done fairly easily; getting $150,000 for new equipment may be 
possible but only over the long-term. 
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He wants to decide now on what to do, but he simply does not know 
which course of action would be the best to take. No matter what 
he decides, he will either be spending — or losing — money, now 
and/or in the future. 

His decision depends, in large part, on the evaluation of these 
proposals. 

The Probl ,..1 of Choice 

Bob Johnson knows that he is paid to make decisions. That's the 
job of executive management. Part of the premium paid to them 
recognizes that their decisions, in particular, are often based 
on a lot of risk and uncertainty. Bob is feeling that risk and 
uncertainty a lot right now. 

Indeed, the driving rhythms of choice and risk intermix into the 
overture that introduces the need for evaluation. Naturally 
enough, people making decisions wish to reduce their uncertainty 
and all the risk attendant to that uncertainty. "Evaluation" is 
a way to reduce the risk of making decisions by generating infor- 
mation and opinion on the value of wnat is under review. 
Evaluation is the hand-maiden to choice. When there is little 
risk or uncertainty — that is, when a decision about what to do 
and how to act has already been made, there is little need for 
evaluation . 

Managers are held accountable for the financial resources 
entrusted to them. This applies as much to the County Extension 
Director's control of his or her budget as it does to a private 
executive's prof it-and-loss-sta tement . Literally, being respon- 
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sible for budget dollars "up's the ante". Managers want straight 
forward, preferably numerical answers to their basic question: 
What is the best course of action to take? 

One ascendent technology that provides hard and neat answers to 
that question is the family of analytical methods known as 
systems analysis (Doughty et al, 1978:3). Harry Summers 
(1981:28) talks about how systems analysis came to dominate 
Defense Department thinking in the 1960 's; he attributes our 
strategic defeat in Vietnam, in part, to exactly this analytical 
bent. He quotes that for such methods of decision-making, " 'the 
catechism is familiar: objectives, criteria, options, costs, 
benefits, quantify as much as possible, focus on changes at the 
margins. ' " 

Systems analysis forces a rationality, an output-and-outcome 
calculus, on events for the purpose of evaluating them in order 
to decide what to do. All forms of analysis usually focus on 
five distinct parts: goals, alternatives, impacts, models and 
decision rules. To reach desired end-states (goals), there are 
always alternative ways to get there. Each alternative has 
impacts (both costs and outcomes). Determining what those 
impacts will be requires that certain models be used. Once 
determined, there will be some criteria (decision-rules) used to 
decide which option to select. Analysis is the prelude to judge- 
ment (Doughty, 1979:7-8). 

Two prominent members of this systems analysis family are cost- 
effectiveness analysis and cost-benefit analysis. Both apply a 
rational yardstick to various alternative courses of action. 
These yardsticks permit a "rational" comparison between the 
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options so that the best choice emerges from the analysis itself. 
Both methods begin by computing the costs of the proposals. The 
costing methodology is derived from basic accounting procedures 
(plus some), is in many ways the simpler part of the equation to 
compute and is the same technique for either method. 

Effectiveness is a constructed measure of how well any given 
program produces the same desired outcome. For example, how well 
do three different teaching systems produce the same commonly 
defined learning outcomes? Some arbitrarily defined index of 
learning is constructed; this index is applied against each 
system to decide that system's relative effectiveness. 

Benefits are the actual dollar values created by the program. 

For a crisis counseling programs, the benefits might be the 

savings resulting from reduced police and ambulance calls to a 

home, and the gains at work for fewer sick-days taken. 

Tne intent of both cost-effectiveness analysis and cost-benefit 
analysis is to provide a means for comparing alternative ways to 
reach a goal(s). Either way, the rationality of the process 
pushes the choice to where gain is maximized at the lowest 
possible cost. 

We'll look at how costs are treated first. Tnen, we'll look at 
the effectiveness issue and explore cost-effectiveness analysis, 
we'll then move into looking at the benefits side of things. The 
conclusion will compare and contrast the two approaches. 
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COSTS OF A PROGRAM 



Begin by determining what your "cost-center" will be. Programs 
can be sliced into at least five distinct cost-centers. At the 
macro-level f there can be th e agency (program umbrella group) 
itself, community Action Agencies would be an example of this. 
Second, there could be a departmental center. These are spe- 
cially organized sub-units of a larger organization. An entire 
Training Department could be treated as a cost center of a larger 
organization. Third, there are programs , such as Meals-On-Wheels 
or Management By Objectives: these are specific, bounded plans 
of action for achieving certain delimited objectives. Fourth, 
sets of program s may be treated as a cost-center. A management 
improvement set of programs could include these programs: a 
supervisory training program, a career path system, an MBO 
program and an incentive compensation program. Finally, specific 
program elements — a computer-assisted instructional element or 
a set of home-reading assignments — could be isolated for cost 
analysis (Beilby, 1979). 

The first task in cost analysis is to decide what the cost- 
center of analysis will be. 

Next, the costs themselves must be determined. A cost is more 
than the out-of-pocket dollar expenses of doing something. 
"Costs" can be thought of generically as all those sacrifices 
associated with a given course of action (Levin, 1975). 
Rothenberg (If'S) refers to costs as those current gratifying 
opportunities that are foregone. When you take a current dollar 
of income (revenue) or a current hour of time which you can spend 
on something you prefer to do now, and instead spend either that 
dollar or that hour for some future outcome, that is a cost. 
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Thus, there is a sublety — and a rigor — to computing costs. 
To simply total receipts is insufficient; all the current gra- 
tifying opportunities must be identified, then their costs esti- 
mated. At the extreme, this could mean including the costs of 
client-time. The second step in cost analysis, then, is to list 
all the specific costs (resources) required for a program 
(Levin, 1975). 



The Economics of "Cost" 

For economists, costs arise because prized results or goods are 
in scarce supply. Goods that are unlimited (such as air) are 
free. Economies arise to produce and distribute these scarce 
goods. The price of a good is its cost based upon the relative 
supply and demand of the good (Gwartney, 1980: 404-419). 

For economists, there are three main kinds of costs. Fixed Costs 
are those established expenditures on factors in production that 
do not vary with the amount of production. Property taxes or 
insurance premiums are example of invariant fixed costs, 
va riable costs are those expenses that change with the amount of 
production (such as wages paid or purchase of supplies). Both of 
these explicit costs can be contrasted with the implicit o^gor- 
tunity costs of action: When resources are used for program X, 
they cannot be used for program Y. 

Add together fixed and variable costs and you get the total cost 
of a project. You can also compute both average costs and margi- 
nal costs as well as unit costs. Sunk costs are those expen- 
ditures, fixed or variable, already spent on the project. Over 
the short-term, costs cannot be changed, essentially. 
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Over the longer term, though, the organization has the ability to 
reconfigure itself so that it can exert some measure of control 
over its costs. Finally, as the 
chart at right shows, costs will 
vary with the output of the project. 
The average fixed costs (AFC) will 
decline with more output. Average \^ 
variable costs (AVC) will initially ^ 
decline but then increase. Average 
total costs (ATC) will reflect the 
component changes. 

QuT Put 

This framework is helpful for understanding the costs of 
programs, because programs will have both explicit and implicit, 
fixed and variable, unit and sunk costs. Cost analysis must take 
these costs schemes into account in order to create a full ren- 
dering of a program's costs. The cost categories should be 
comprehensive to include all the resources consumed by a project 
or program. 




Determining Program Costs 

The following discussion will show how to compute the costs of 
programs and functions. While the model could be applied to 
departments or entire agencies, the accounting for those cost- 
centers are sufficiently distinct in treatment. Finally, the 
referent example will be training programs, but again, the ideas 
discussed should be sufficiently applicable to all kinds of 
programs. 

A program's fixed costs could include expenditures in research 
and design, equipment and software and facilities (if 
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applicable). Variable costs would be trainer time (in and out of 
the classroom), plus any pro-rated expenses for administrative 
overhead, supplies, etc. The opportunity cost is the 
trainee' s/participant* s time spent in the program. 

Weinstein (1982) summarizes th':se various factors into four cost 
categories. 

1. Direct Program Expenses all expenses directly attribu- 
table to a given program. 

2. Administrative Expenses all administrative overhead 
expenses related to but not directly attributable to a 
given program (such as the pro-rated salary of the 
Department Director, etc.) 

3. General Organizational Costs — such pro-rated expenses 
incurred by the larger sponsoring organization in managing 
the program (in accounting, person; el, executive manage- 
ment) that are not directly related to the given program. 

4. Participant Expenses — the total compensation paid to the 
employee while involved in the program, plus any related 
on-the-30^ time spent on the training. 

The chart of accounts used to classify costs could involve such 
specific items as personnel, hardware, software. See Appendix 1 
for a suggestive listing of such items. What is important is 
that a comprehensive, easy-to-use chart of accounts be employed. 
Ag-'in, all relevant costs of the program must be accounted for. 
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Indoed, it is possible to organize the above two options into 
one: make an exhaustive listing of specific items (wages, bene- 
fits, supplies, hardware, software, etc^) and them compute what 
each of those items would be for direct, administrative and 
general organizational expenses. The resulting table shows this 
chart-system of cost accounts. 



Example; Account Charting 



Cost 
Group 



Line 
Ii-ems 



Direct 




General 


Program 


Administrative 


Organizational 


* Salary 


* Salary 


* salary 


* Supplies 


* Supplies 


* Supplies 


* Equipment 


* Equipment 


* Equipment 


* Facilities 


* Facilities 


* Facilities 


Etc. 


Etc . 


Etc. 



Finally, to develop the proper cost model, you must also know 
about project's life-cycle. Programs evolve. The nature of 
costs will vary over time. Kearsley suggests four distinct phases 
in a program's life-uycle. First is the analysis, R&D design 
phase. This is followed by start-up, investment phase. Once in 
place, the program begins an operating, maintenance phase. One 
program may be replaced by another; during this transition 
period, a new cost structure will take place. 
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The following map to pigeon-holing a program's costs emerges. 
Line 



Total 



Direct * Salary 

* Supplies 

* Hardware 
Etc. 












Adminis- * Salary 
trative 

* Supplies 

* Hardware 










$ 


General * Salary 
Organi- 
zation * Supplies 

* Hardware 
Etc. 










$ 


Participant * Compensation 
Rate 










$ 



Total 



Such a cost computation is necessary for a full and complete 
estimation of program costs. It also suggests the timing of the 
expenses and the relative proportions of expense items. This 
generic cost-map would be modified to meet the unique specifica- 
tions and dimensions of each program. 

Re counting Standards on Costs 

A discussion on program costs would not be complete without some 
reference to the necessary accounting standards wnich should 
apply. 
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Of course, the critical principle in cost accounting is that 
the asset or expense item should be recorded at its real cost 
(Meigs, et al: 16ff). Tnis means, as much as possible, using the 
actual paid price (the historical cost) of the item. This price 
is not what the asset could be sold for now nor does it indicate 
what would have to be paid in order to replace the iten. For 
physical equipment, facilities, supplies and other tangible 
objects with a market value, this cost principle is easy to 
establish. It is when various "subjective" costs (such as client 
goodwill) are considered that this cost principle becomes much 
more difficult to follow. 

Beyond this basic principle, costs have other implications, 
including: 

1. Amortization of Capital Items . 

capital items are typically pieces of equipment or other 
tangible goods used in producing outputs? their "life" 
extends longer than one year. The outlay cost of such 
items should be spread, usually equally, over the esti- 
mated life of the object. Since a capital item may be 
used by more than one cost center, its expense must be 
pro-rated, based upon the cost center's use of it 
(Beilby, 1979). 

2. Program Length . 

Some program cost-center's exist for longer than one 
accounting, budget cycle. Multi-year projects require 
multi-year cost estimates. A complete rendering of future 
year costs should be done using constant dollars. This is 

HR"0"/54(24)sv 

ERIC 



turn means computing the present values of future dollars. 
(See Weston et al, 1982: 49-63, for a discussion of pre- 
sent value. ) 



3 . Matching Costs & Outcomes 

You should use the same period of time to measure both 
costs and outcomes. Therefore, if you decide the program's 
length will be two and one-half years, count the costs 
incurred during that total 2 1/2 year time, but also count 
the benefits/results accruing for that total 2 1/2 years, 
at least. (There are other considerations, too, about com- 
puting results? see the discussion below.) 



4 . Ranges of Uncertainty . 

Certain cost estimates can be done with a high degree of 
certainty? some, with a high degree of uncertainty. 
Should the uncertain figure be a large item in the cost 
computations, the resulting total cost would be subject to 
the same uncertainty. Therefore, apply a "sensitivity 
analysis" to the cost estimate by using upper and lower 
points; factor these differing points into the total 
equation to see how the cost estimate can vary 
(Levin, 1975:95). 



5. Sunk Costs. 



Do not include sunk costs as part of your cost estimates. 
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Displaying Costs 



Finally, the costs of a program can be portrayed in different 
ways. AS the life cycle - chart of accounts model above 
suggests, costs can be broken into specific line items (salaries, 
supplies, equipment, etc.), into cost groups (direct, overhead, 
etc.), or by the life-cycle phase. Weinstein (1982) suggestsa 
participant learning hour as a unit cost for training programs. 

Levin (1975:98) suggests three additional options: 

1. T otal program costs can be used to good effect whenever 
the various alternatives yield about the same result. As 
Kearsley (1981: 54) is quick to note, selecting the best 
program here is easy: when the results produced by the 
alternative programs are about the same, pick, the least 
expensive one. 

2. Average unit cost of result : when output levels differ by 
program, a "unit cost" can be very helpful in comparing 
the programs. Be careful to keep in mind the absolute 
levels of output, though: don't let an average unit cost 
of $100 (but with a total effectivness of only, say, 175) 
confuse you if a second option average 125 but with a 
result of 750. 

3. Marginal unit cost of resu lt: use this wxien unit costs 
change rapidly with the volume of the program. 

Costs; Summary 

Here, then, is the summary on how to compute the. costs of a 
program. 
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1. Define your cost-center: entire agency? department? 
program? set of programs? program element? 

2. Identify all the resources consumed by the program. The 
life cycle-chart of accounts model can focus this listing. 

3. using the cost principle, compute or estimate the costs of 
each resource. 

4. Decide which costs you will include and which you will 
exclude. 

5. For ''big ticket items" whose costs are uncertain, complete 
a sensitivity analysis with a range of values for the 
items . 

6. Show either total costs (which can be subdivided into 
line-item, functional or life-cycle parts), average unit 
costs or marginal unit cost. 

This process is to be used for either cost-effectiveness or cost- 
benefit analyses. Program costs are one side of the analytical 
process; finding program outcomes is the other. We will now look 
at one way to capture those outcomes: effectiveness. 
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IDENTIFYING RESULTS 



Programs are instituted with the intent of producing results: 
more learning or skill, better capability, less expense, more 
revenue, whatever. The relative ability of different programs to 
produce these results — compared against the costs of each 
program — is the essence of rational cost-result analysis. 

There is some confusion, though, in this process. As noted 
above, the cost side of the analysis is reasonably straighfor- 
ward, albeit subtle and involved. Where things become dif- 
ficult is on the results side, and part of the entire problem is 
confusion over what "results" are. "Outcomes", "returns", 
"benefits" , "outputs", "results", "products", "ef f ectivenes|^-- 
all are part of the vocabulary. The problem is that there no 
consistently used definitions to these terms. For example, its 
almost equally likely to look at a study of "cost effectiveness" 
(Cullen, et. al. 1978) to see that it really is about financial 
returns or to look at a "cost benefit" analysis (Kearsley, 1981; 
1982) to discover it is more concerned with effectiveness. 

Let's try to clarify the terms. Programs are designed to create 
outputs or products. (I will use the terms "output" and 
"product" synonymously.) Products are the outputs of the work of 
the program and can range from "welfare payments" to a "trained 
employee" or a "better functioning group" (or family or team). 
Programs are organized work-methods that take inputs and recom- 
bine those inputs to make specific outputs or products. The pro- 
duct or output is what ends up after tje]}/ program' s work-process 
is completed. 

These products (can) have some monetary value. The value of a 
welfare payment is at least the amount of the check. Even a 
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"tramed employee" can be valued (although the valuing method is 
a little more involved; see Casio, 1982). When the program 
creates these products for use in the private marketplace, these 
values can accrue to the firm. An dbvious example would be a 
managed swimming area at a State park that requires a fee for 
admission, or a sales training program. Both "programs" generate 
an additional stream of income for the organization. When done 
for the public sector, the value accrues to the larger community. 
Here, free energy counseling would be an example. Let's refer to 
the revenue produced by a program as a "return". The monetary 
value created by the program is the "return" of that program. 

When dealing with costs and returns, there are three traditional 
methods for analyzing them. The easiest method is the payback of 
the program: how long will it take the program to generate a 
return equal to the cost of the program? An incentive program 
costs $10,000 "up-front", and it will take nine months before the 
outputs of this incentive program (more work, less expense, 
etc.) equal to a return of $10,000. The Payback Period of this 
incentive program is nine months. Second, it is possible to com- 
pute the return on investmen t (ROD of the program (in finance, a 
program is called an "income-producing project"). Consider the 
total costs of the project as the investment in it and then 
divide that into the total expected return. A job enrichment 
program requires a total investment of $25,000, and the expected 
return on it (over the project's total life) is also $25,000. 
The ROI of this project is unity (1). Should the return be esti- 
mated at $50,000, the ROI would equal 2; a $100,000 expected 
return would be 4 — for every one dollar spent, four would be 
collected. Finally, a program's internal rate of return shows 
the equivalent yield of a program in the organization's internal 
accounting system. (See Weston, 1982 for a fuller discussion.) 
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programs have outcomes, though, which are more than the sum of 
their intended return parts. Cooperative extension programs do 
more than simply yield more profitable crops; they serve as com- 
munication links, build an attitude of innovation, act as sup- 
ports to farmers, etc. (Baer, 1977). 

These kinds of outcomes of programs have to be considered, too, 
when defining what a program does. These outcomes may be 
impossible to quantify, though. Since decision-makers need a 
full accounting of what a program makes happen, "outcomes" can 
simply be listed and included as part of an evaluation or analy- 
sis piece. Clark & Olsen (1977) suggest a method for showing the 
outcomes as part of the analysis: using a "T-account " model, 
simply list the outcomes in one column and the degree of impact 
(high to low) in the adjoining column. 

Finally, there is effectiveness, and there are benefits. 

As is often the case with programs aimed at affecting human 
beings, the results directly created by the program are difficult 
to conceptual ize as products or outputs. The learning created 
by a training program or the improved health practices resulting 
from an outreach program or the altered home energy conservation 
measures taken by families as a result of a counseling program 
are difficult to fit into a "product" mold. These results are 
less tidy and cannot be formed into distinct, repetitive, stan- 
dardized outputs. Yet results do occur. The problem is how to 
create some reasonably consistent measure of program results. 
This is where effectiveness comes in. (We'll pick up on benefits 
below in that section.) 
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EFFECTIVENESS" ANALYSIS 



A program's effectiveness is the program's ability to produce or 
yield an established result. This result is defined in terms of 
a constructed (that is, artificial) measure* It is a single 
measure of results based upon the intended objectives of the 
program. These results, as indexed by the effectiveness mec rre, 
are not put into a dollar value (Forbes, 1974; Lent, 1979). 

Cost-effectiveness analysis, then, compares the costs of a 
program against its power to yield a level of results on some 
single measure of effectiveness. There should be at least two 
alternative programs considered. Cost-effectiveness analysis, 
thus, is an analysis of the extent to which competing programs or 
techniques create a desired output. The decision-rule is basi- 
cally to select that option which produces the most yield at the 
lowest cost. Cost-effectiveness analysis assumes a constant, 
given goal. It then proceeds to fashion an answer to the 
question: "given output X, which program A^ B,... N will produce 
the most of X at the least cost?" 

The measure of effectiveness, thus, must be a result that is common 

to all the alternative programs. A, B N (Levin, 1975). Given 

the goal of a trained employee or a health-practicing individual 
or an energy-conscious family, there can be at least two dif- 
ferent programs used to produce that goal. An effectiveness 
measure is constructed to determine how well each of those dif- 
ferent programs yield those results. The actual measure of 
effectiveness can be constructed any number of ways: 

* For a training program, say, with well-defined learning 
objectives that can be summarily assessed with one paper 
and pencil test, the effectiveness measure could be test 
score results (Forbes, 1974). 



HR"0"/67A(28)sv 

ERLC 



* For programs whose results may be more complex (such as a 
heaith-practices program), results could be multi-faceted: 
various kinds of knowledge attitude and behavior actions 
that together produce improved health-practices by the 
individual. In these cases, some sort of combined,, 
weighted measure can be constructed to assess the various 
program results. Thus, the single health-practices measure 
could be constructed as such: 



* score on health knowledge test x .25 = 

* score on health attitudes survey x .40 = 

* score on health behaviors 

self-report questionnaire x . 35 = 

health-practices measure 



The weighting (here, ,25, .40, .35) given to each factor is in 
many cases a subjective hunch or guess as to the relative impor- 
tance of each sub-factor in producing the total outcome. Thus, 
by this method, "attitudes" play a slightly larger role in 
health-practices th^hn "behaviors" and both together are three 
times mo^e important than "knowledge". 

Of course, the best way to decide what the actual effectiveness 
results are is to use the actual program results. This presumes 
an experiment using the different programs with the effectiveness 
results of each collected and computed. For after-the-fact eva- 
luations, this is the preferred approach. When not possible, it 
may be possible to use a panel of experts to assign effectiveness 
scores to alternative programs (Shipp, 1980). Or it may also be 
possible to use previous empirical or normative data to estimate 
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relative program effectiveness, too. These last two methods may 
be the only options to use if experiments cannot be performed or 
if cost-effectiveness analysis is being used in planning. 

Effectiveness Analysis; Steps 

There are three basic steps to complete when doing an effec- 
tiveness analysis (Lent, 1979), 

First, identify the alternative programs to be compared. This 
assumes, of course, that the goal has been established. The 
decision-maker/evaluation sponsor may tell you what the different 
programs are; if not, brainstorm a list of alternatives. 
Describe how each of these program alternatives operate to pro- 
duce the established program results. 

Second, in conjunction with the decision-maker/sponsor, establish 
what the effectiveness measure will be. Because this measure is 
artifical and indirect, it is important that the user of the data 
understand and agree to what is being used and why. 

Third, actually design and construct the effectiveness measure. 
This could be a simple as validating a 10-item test or as complex 
as designing various data-collection methods, pre-testing them 
and then cranking out a combined, weighted measure. 

Once the actual measure is developed, data can be collected and 
the effectiveness score computed. 



HR"OV69(21)kk:sv 



^2 



-4- 



Ef fectiveness Analysis; Caveats 

There are a couple of cautions to keep in mind when doing an 
effectiveness analysis. For example^ such an analysis should 
never be done if the cost of doing it is more tnan the cost of the 
least expensive, wrong alternative. 

A program's results may be spread unequally across the target 
population. Using a single measure (for example, one 10-item 
test) may not be that sensitive a register of the program's 
effectiveness. Reliance on a single criteria could be a strate- 
gic evaluation and analysis mistake. 

Likewise, with multiple registers, the weighting scheme could 
skewer the effectiveness scores. Such weightings are based on 
best-guestimate estimates — but are still personal judgments. 
It may be necessary to use different weighting schemes in order 
to determine the internal validity of the process. 

Another caution concerns using inappropriate measures. Doughty 
(1979: 23) warns about using input or through-put indicators 
when output indicators should be used. Computing a class-contact 
hour's effectiveness measure will not tell anything about the 
results of that time. 

A generic issue involves the potential decision-maker myopia 
which cost-effectiveness analysis incubates. One way this occurs 
is through a one-sided, exclusive reliance on numbers — of costs 
and of effectiveness. This myopia also blurs the decision- 
maker's ability to see unanticipated, spill-over, collateral or 
uncountable effects of a program, and then take them into account 
when making a decision. 
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Cost-Ef fectiveness! E xamples 

Two examples can show cost-effectiveness analysis in operation. 

As a result of legislative initiatives in 1977, Congress wanted 
more people served by the Expanded Food & Nutrition Edt'.cation 
Program (EFNEP). A joint experiment was conducted in Vermont and 
Nevada (Honnold et al, 1980) to test the cost-effectiveness of 
three different educational delivery systems. The goal was to 
significantly increase the nutritional practices of a selected 
target population. The population was food-stamp recipients with 
no prior EFNEP experience who were assigned to various groups in 
the 3-month study. 



follow-up. Program 3 was the traditional n^^.^^^^^^^^ 
direct teaching method (one to one or small 
group) . 

A learning effectiveness measure was created. This measure was a 
weighted composite index combining test results on nutritional 
knowledge, food recall and nutritional behavior. Pre- and post- 
test scores on this measure were obtained from the various groups 
who were exposed to the different delivery systems. (Other 
aspects of the programs — coverage, costs, efficiency — were 
also studied. ) 
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AS the chart at right shows, there 
were three educational programs tested. 



method. Program 2 did not use any phone 



program 1 was a combined TV series, 
direct mail and telephone follow-up 
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The result? While the learning effectiveness of the three were 
about equal, costr were not. Program 2 was clearly the raost 
cost-effective? Program 3, the least; and Prograjn 1 was about 
half-way between. 

At Florida State University, the traditional large lecture, sur- 
vey course in Geology was converted to a self -instruct ional 
system. Costs were computed using the life cycle and chart of 
accounts method suggested above. The effectiveness measure was a 
scaled pre- and post-test computation from both programs. The 
individualized system did increase both effectiveness and costs. 
Two other programs — small-group instruction and a commercially 
developed self-instructional package — were added in for com- 
parison. When this was done, with higher levels of effectiveness 
and the lowest unit costs, the commercially developed self- 
instructional program was the clear favorite. (Doughty and 
Stakenas, 1973. ) 
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THE BENEFITS OF SOCIAL ACTION 



Imagine spending $150,000 on equipment... or $150 million on a 
dam. Either way, for the decision-maker involved, that will be 
a lot of money. That decision-maker will want to know what he or 
she can expect to get back from that outlay of funds. The same 
issue faces decision-makers wondering what they get back from a 
quality circles program that costs $20,000 or a public health 
service that costs $20 million. The key issue is the same; 
••What do I get back?" 

Cost-benefit analysis is a way to answer that question. In this 
sense, cost-benefit analysis is a kissing cousin to the return on 
investm.ent technique noted earlier. Unlike the measure of effec- 
tivness, a benefit is described in monetary terms. In this way, 
the two factors in the comparison — cost and benefit — are 
expressed in the same terms, making the actual comparison much 
easier (Temkin, 1974). Cost-benefit analysis, thus, offers an 
additional way to evaluate a program. 

cost-benefit analysis is traceable to the laws of the early 
1900 's governing river and navigation projects of the Army Corps 
of Engineers. For example, the River & Harbor Act of 1902 
requires the Corps to report on "the costs and benefits" of their 
various projects. Later, this reporting was added to flood 
control projects (Prest & Turvey, 1965: 6&3ff). Today, cost- 
benefit analysis is often used at the macro-economic level in 
evaluating policy and program issues, such as whether it is 
better to build a highway, add a hospital or engage in a popula- 
tion-control program. For developing countries, these issues are 
critical; when resources are scarce, getting the correct answer 
is essential (Rothenberg, 1975). 
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This basic analytical process can be applied to projects and 
programs less grand in scope, though. We'll look at the mecha- 
nics of that process, then see the process at work in a few 
examples . 

Defining the Benefits 

Programs create changes in people (at least, that is their 
intent). Often, these changes lead to real, monetar ily-def i- 
neable outcomes in the lives of the people in the program or 
others indirectly affected by the program. As Rossi (1982: 268) 
points out, such changes are net, marginal outcomes. These 
marginal benefits are expressed in dollar value. When compared 
against program costs, the decision-maker can see what return the 
expenditure creates. 

The actual steps in a cost-benefit analysis are straight-forward; 
remember that the cost side has been reviewed earlier: 

1. Trace out the expected outcomes that the program will 
produce (or has produced). This list will include both 
hard and soft, objective and subjective oui:comes. One 
way to do this is by thinking through what changes the 
program produces in the target population; this will 
either be adding behaviors or taking behaviors away. 
Either way, list out all the specific results those 
changes will produce. For example, a job search workshop 
may accelerate the speed with which a person finds a 
30b. Specific results might include: 

* Marginal increases in income (new regular salary minus 
previous unemployment compensation) . 

* Less stress-induced illness. 
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* Retained possessions (furniture, car, savings), 

* Enhanced self-esteem. 
The list could go on. 

2. Select those results whose value can be put in monetary 
terms . In many cases, this may be more of a tactical, 
analytical decision: can reasonably meaningful monetary 
data be generated? The intangible benefits will be 
reported too, but in the narrative of the evaluation. 

3. Identify the control group . Programs should make a dif- 
ference, and it is in that marginal difference that bene- 
fits are computed. Therefore, a base-line of non-program 
performance must be established in order to find out what 
the people would be like "normally" without the effects 
of the program. For example, how much stress-induced 
illness is caused by unemployinent , and what is the cost 
of that illness? Or, what is the typical unemployment 
compensation received by the target population? 

4. Describe the effects of the program . Here, the extent to 
which the program produces either direct or indirect 
changes mvst be detailed. This is the leverage or amount 
of change produced. It is in this marginal impact that 
benefits are computed. Using the unemployment example, 
baseline data might show that workers were unemployed an 
average of nine months. Results might then show that 
attendance at this workshop lead to a job within two 
months, and that unemployed individuals would invariably 
sign up by their third month of unemployment. The net 
change would be to produce employment within two months; 
the net effect, four less months on unemployment. 
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5. Decide on the duration of the benefits . As we'll see in 
the example below, some programs (example: Upward Bound) 
create benefits that last the participant a life-time; 
others last only as long as the person is in the program. 
This decision is required in order to determine what the 
full benefits of the program are. 

6. Tabulate the total monetary benefits . At this point, you 
are working with variables that can be put in monetary 
value; you have both baseline data and data on program 
changes, plus a decision on program duration- Using a 
table, compute the estimated monetary benefits of the 
changes over the duration decided on. 

7. compute the present value of these benefits . Tenkin 
(1974; 41) shows the standard present value formula, 
applied to these variables: 



\ = 



N 



(Bt ^ Ct) 

(1 + i)^ 



where 



t=l 



V = the present value of project A. 
B = benefits of the program 
C = costs of the program 

t = time period (for example, 6 month segments) 
and N = duration or number of years of the project's use- 
fulness. 



8. Sum the component present values in order to reach a 

total benefit of the program (in current dollars) . Then , 
co mpare benefits against costs . 
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Like all forms of analysis^ there are certain decision rules 
which apply to the final comparison. 

The guiding rule is to select programs that give a maximum bene- 
fit at the lowest cost. Generally, this will mean selecting 
those programs where: 

* Present value of benefits exceed present value of costs. 

* The annuity flow of benefits exceeds the annuity flow of 
costs . 

* The internal rate of return exceeds the discount rate. 
(Prest. & Turveyr 1965: 715.) 

Caveats 

Of courser the actual analysis is never that easy. The items 
below are some of the issues and cautions to watch when doing the 
analysis : 

* AS in all research, look at how well your sample of program 
participants represents a larger target population. The 
best analysis in the world will be for naught if the sample 
is too idiosyncratic for larger comparisons. This means 
that you must identify the key dimensions of your sample 
while knowing the key dimensions of the larger universe, 

* As noted earlier, use real market prices. Express all 
monetary values in constant {present value) dollars. 

* Evaluate the validity of your data. Data, here, will be of 
two kinds: market prices usea and program effects • At 
times, either or both of these may not be clearly, defined 
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or established — or if they are, their applicability to the 
current evaluation is indirect. Point out whether, in your 
opinion, the data will lead to an over-estimate or an 
under-estimate of real program effects. 

* In a related matter, make every effort to base your data on 
empirical findings. Minimally, scour the literature to see 
what kinds of effects similar programs have. As much as 
possible, use that information as a basis for defining 
impacts. Pulling numbers out of a hat should be the last 
option — and may not be appropriate even then. 

* You should try to compute all effects which lead to 
material changes, either directly in the lines of program 
participants, or indirectly in the lives of others. Price 
changes caused by the program can be excluded. (Thus, in 
the metadone program described below, the changes in the 
price of heroin caused by the supply of methadone was not 
included, while the savings in criminal justice system pro- 
cessing were. ) 

* Compute benefits for the realistic length of the program's 
effect. This length or duration can be based upon best 
guess or empirical data. Keep in mind, though, that there 
may be real outer limits on a program's duration, caused by 
such things as: technological or economic changes (in our 
job search example, that might reduce the number of 
employers in an area)? maturation of the population (where 
most participants simply outgrow the use of a drug); or 
other "secular drift" kinds of trends. 

* Tne discount rate used should be selected carefully* 
Generally, prevailing market rates can be used. 
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* Apply a sensitivity analysis to key variables. For 
example, in periods of fluctuating rates, two or three dif- 
ferent discount rates might be used. Or, for certain bene- 
fits that are based on hunches, a range of values might be 
computed. Likewise, the value of a questionable benefit 
might even be excluded in a second computation. 

* As a general rule, be conservative in estimating benefits: 
always go for the lesser, more conservative number. 

* As noted, intangible benefits that cannot be monetarized 
can be included — in a narrative statement. 

* Finally, include any qualifications or underlying assump- 
tions. These are the premises on which the computations 
were based — such as that the unemployed person was indeed 
looking for a job and that the number of employers in the 
area held constant. And, in the final analysis, what may 
be more important is the durability and predictability of 
the benefits, not the actual number itself. Thus, in 
making a decision about the job search workshop, the 
decision-maker may want more assurance about the connection 
between attendance and finding a job than about the exact 
dollar value of the benefit (Roid, 1974: 61). 

Cost-Benefit Analysis: Examples 

W. I. Garms (1971) looked at the costs and benefits of the 
Upward-Bound program for disadvantaged yet capable teenagers; 
this program was intended to move them into a college preparation 
track. Using as a control, baseline group their older siblings, 
a sample of over 7,000 program participants was examined in terms 
of their high school completion and college entrance rates. 
Benefits for life-time income, a program stipend and scholarships 
were compared against such costs as college tuition and foregone 
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earnings while in school. These computations were done from the 
individual's viewpoint as well as from the larger societies point 
of view. An example of the former is shown below for non--white 
males; note that both a 5% and 10% discount rate was used. 



Non-White Male 



5% 10% 

Benefits (Total) $4,850 $1,129 
Costs (Total) 916 

Net Result $3,934 $ '371 



At both the 5% and 10% discount levels, the value of the benefits 
per participant exceeded the per person cost. Gai'ms concludes 
(p. 220): 

From the economic viewpoint, upward bound is at best a margi- 
nal program, and the justification for its continued existence 
must be sought in presumed benefits which are not accounted 
for here. 

T. H. Hannon (1976) looked at the costs and benefits of a New 
York City Methadone Maintenance program. Benefits from reduced 
heroin use accrue to the users themselves ( in terms of better 
health, les arrest, saved expense of heroin and possible legal 
earnings), to the taxpayers (with less criminal justice system 
and health systems expenses) and to the potential victims of 
users (less expense for private police protection, property 
damage, less fear and anguisn). Since some of these are uncom- 
putable, Hannon computed the benefits of savings due to reduced 
criminal justice and health expenditures as well decreased heroin 
expenditures. The control was their pre-program behavior. Costs 
were computed for a residential program and for an out-patient 
one. At a 10% discount rate, benefits were always greater than 
costs. 
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CONCLUSIONS 



When faced with limited resources, decision-makers feel pressure 
to select the best course of action from among competing alter- 
natives. Wnile any method of decision-making could be used 
(including reading a horoscope or the entrails of certain 
types of fowl) , managers tend to prefer rational analysis that 
give them some indicator of what they will get for what they have 
to pay. 

Cost-ef ectiveness analysis and cost-benefit analysis are two 
related, yet distinct approaches to solving this problem. For 
both, costs are computed similarly. Various program line items 
are computed at cost for the various phases of the program or 
project. These costs may be reduced to constant (present value) 
dollars for multi-year programs and certain parameters may 
altered to show differing cost results. 

The gains of a program can be described in many ways. 
••Ef fectiveness** is typically an indirect, constructed measure of 
selected program outcomes. The measure itself is some sort of 
test result or behavioral score on a yardstick applied to com- 
peting alternatives. The outcome is held constant; each alter- 
native has the same effectiveness yardstick held up to it for 
comparison. Thus, for the same unit cost, one can determine 
which option yields the most effectiveness; or for a given level 
of effectiveness, which program produces it at the lowest cost. 

"Benefits'* are the outcomes of a program expressed monetarily. 
These are all the quantifiable returns created by the program. 
These returns may accrue to the individuals participating in the 
program and/or to the larger society (including taxpayers). 
Because these returns are not necessarily collected by and hence 
recorded into the accounting system of the institution .sponsoring 
the program, these returns are called "benefits". 
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Both techniques provide ways by which decision-makers can judge 
the options before them in order to render a decision about the 
best course of action to take. They both can be applied to eva- 
luations done ex post facto or to pre- implementation , planning 
evaluations. The quality of the actual analyses depends upon the 
quality of assumptions and data used. The refrain •'Garbage In, 
Garbage Out" is sung here, too. Without some empirical 
grounding, such analyses based upon speculations have limited 
utility. Thus, when either of these methods are used in the 
planning phase and are based upon speculative hunches, their 
total value in decision-making should drop correspondingly. 

Both methods, then, are ways to compare competing claims on 
resources. The rule for evaluating and deciding is essentially 
the same in each case: select the option(s) that yield the most 
at the lowest cost. This is a guideline , for it is probably never 
possible to run across the program that simultaneously has the 
best outcome and the lowest cost. 

In cost-effectiveness analysis, the desired outcome is the same. 
Different ways to produce that outcome are compared. The time 
period required to produce those results is not considered 
though. All the alternatives are examined from the point of view 
of what they share in common: a way to produce a single outcome. 

Cost-benefit analysis, on the other hand, allows the decision- 
maker to compare different programs with different goals. This is 
because all parts of the analysis— -the costs and the benefits of 
all the different programs — are reduced to one common yardstick: 
money. The cost-benefit ratio is the means for comparing the 
options . 

The constant yardstick in effectiveness analysis is the desired 
outcome (as indexed by an effectiveness measure). For benefits 
analysis, the constant yardstick is monetary value. 
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AS a final reminder, decision-makers may find either analysis 
helpful. But they should not use the results as an excuse for 
their decisions. Decision-makers are paid a premium to make 
riskier decisions. Judgement is different than formula. Neither 
of these methods produce judgement; they merely execute certain 
formulas. Therefore, neither method is a panacea to the problem 
of choice. They are assists. The decision-maker must remember 
to use judgement. This means relying on data produced by these 
analytical techniques. But it also means not being blinded by 
them. Problems of choice must be dealt with using all the infor- 
mation available — and this will include the intangible, subjec- 
tive and non-quantifiable, too. 
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APPENDIX 1- COST CATEGORIZATION FORMAT 
Examples of Activities and Categories for Cost Analysis 



Taken from Doughty & Stakenas, Accountability; Systems Planning 
in Education, ETC, 197 3. 



I. Activities 

A. Research and Development 

1, Needs assessment - front end analysis - initial 
planning 

2, Task analysis - job analysis 

3, Curriculum design 

4, Prototype development and testing 

5, Formative education - preliminary product and 
program review 

6, Materials validation 

7, Training program and equipment development^ 

8, Initial personnel recruitment and/or training 

B. Investment and Production 

1. Acquisition - Installation - Start up costs 

2. Procurement of initial stock of training hardware 
and software 

3. Duplication of production masters 

4. Construction - Renovation of facilities 

5. Purchase of initial spare components 

6. Modifications of existing systems 

7. Initial deployment of training hardware and software 

8. Initial dissemination of duf fusion and implementa- 
tion activities 

C. Replacement 

1. Attrition 

2. Replacement as a result of: 

a . Obsolescence 

b. Depreciation: Normal use 

c. Theft - Vandalism - Breakage 

3. Periodic (scheduled or unscheduled) updating of: 

a. Content - materials 

b. Equipment 

c. Procedures - management 
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D • Operation 

1. Personnel 

a. Instructional: salary - travel - benefits 
(including retirement) 

b. Administrative - managerial 

c. Maintenance - support 

d. Students: salary - travel - benefits 

2. Materials - consumables 

3. Ongoing training and evaluation 

4. Ongoing distribution - deployment of hardware and 
software 

5 . Facilities 

6. Overhead 

II . Cost Categories 

A. Personnel: Salaries and Benefits 
1 • Instructional staff 

2. Support staff: Non instruction - secretarial 

3. Program administrative - managerial personnel - 
supervisors 

B. Hardware 

1, Simulators - trainers 

2. Audiovisual equipment 

C. Software 

1. Instructional materials and supplies 

2. Training aids 

3. Expendable materials 

4. Training manuals, technical manuals 

D. Facilities 

1. Classrooms 

2 . Laboratories 

3 . Self-instructional facilities 

4. Administrative - managerial - support facilities 

E. Institutional Overhead/Administration 

1. Agency - institutional management 

2 . Libraries 

3. Computer facilities 

4. Contracted services - consultants 

5. Institutional overhead 
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A. C ost Activities 

Program activities are a major focus of any categorization 
system. Several referenced models employ functional category 
descriptors similar to those in the outline, but that is not to 
say that these categories are the only ones that might be accep- 
table. Several referenced reports include sections containing 
extensive specification and definition of program activity cate- 
gories. For our purposes^ a brief overview should suffice. 

1. Research and Development 

Costs include all funds expended to bring an ongoing or 
planned program (alternative-strategy) into readiness for 
implementation. R&D expenditure for front-end analysis, design, 
development, formative evaluation, staff (preservice) training, 
and procurement of R&D materials and equipment are essentially 
one-time non-recurring costs and should be so identified. Some 
non-recurring costs, however, should not be assigned to a R&D 
phase or function but to an investment and production function. 
This second major activity includes all dollar costs required to 
phase a program into operation. These include costs for facility 
renovation or procurement, instructional equipment, acquisition, 
and production/duplication of instructional materials. If 
materials are commercially available, they are also charged to 
this activity. 

2. Facilities 

Many schemes include some prorated estimate of facilities 
or overhead cost in either the investment or operation phase. 
Considerable study still needs to be given this particular area, 
but if feasible alternatives being considered all require similar 
existing facilities, then these costs can conceivably be 
classified as a constant and perhaps be excluded frc«n the analysis. 
This will not be the case if new or additional facilities or 
other overhead are required. Appendix A contains a brief 
discussion of the range of alternatives for costing instructional 
space. 

3 . Replacement 

Many models also include various replacement costs for 
equipment and materials in the operation cost category. This may 
not be appropriate if alternatives being compared differ drasti- 
cally in this area. To include the replacement cost of a com- 
puter or a simulator in the general operating category *may be 
highly misleading. Within this area, predicting depreciation and 
obsolescence rates for instructional materials and equipment is 
often described as an art form. The traditional estimate of a 
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5~to~10 year lifespan for training hardware^ for instance^ is 
based on "normal usage." Factors such as maintenance schedules, 
amount of use, type of use, user sophistication, and theft rates 
should be considered and somehow factored into any replacement 
estimate. Obsolescence predictions for software will depend upon 
such variables as content stability, style changes in visuals, 
format (hardbound, cassette, workbook), and usage. 

4. Operating 

A considerable portion of the recurring costs for any 
program or alternative should be included in the operating cost 
categox'y« Instructional personnel, periodic maintenance, 
expendable supplies, summatAve evaluation, in-service training 
and managerial overhead are all recurring costs and should be 
classified as operational costs. In a conventional setting, one- 
cycle operation costs might be the funds required to maintain one 
program (course) for one complete iteration (cycle). Innovative 
programs will present much more of a challenge. One cycle of an 
individualized, non-time-based, ship-based program is not so 
easily segmented, categorized, and costed. Considerable review, 
testing, and revision will be necessary before any existing con- 
ventional procedures will be useful. 

A predictive model may make use of ex post facto data to 
establish cost estimating relationships. It is therefore impor- 
tant that a general cost summary be suitable for both descriptive 
and predictive data. When lifetime operation costs are required 
(and they are most important, albeit speculative) conventional 
programs present much less of a problem than newly evolving 
innotative types. Although predicting the obsolescence rate of 
course content, the number of times per year a course is offered, 
and the number of years it may continue to be offered (before 
major revision) is a challenge, it does not compare to the dif- 
ficulty of estimating the scope, duration, and number of cycles 
in the "life" of a modularized individualized program. 

Estimating life cycle costs for Navy personnel has been 
greatly simplified by analysts in the Personnel Plans Division of 
the Bureau of Naval Personnel. A sporadically published report, 
"Navy Military Manpower Billet Cost Data for Life Cycle Planning 
Purposes" (which was last published in 1973), contains comprehen- 
sive annual billet costs for both enlisted and officer personnel. 
Included in the reported figures are actual and estimated costs 
for retirement and other fringe benefits. Needless to say, these 
particular costs can be a significant factor when comparing 
alternatives with differing degrees of labor intensiveness . In 
addition. Appendix B contains a discussion of alternatives for 
determining personnel costs for an operational instructional 
system. 
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B. General Guidelines 

A reasonable guideline for after-the-fact or predictive cost 
analysis is to devote attention to any particular category 
according to the proportion of the total budget reflected by that 
component. Obviously r personnel costs in most training and edu- 
cation contexts will account for a large percentage of any budget 
so an analyst's energies should reflect that fact. Fortunately r 
personnel costs hold few surprises or computational difficulties 
and may usually be guided by past cost experience or programs 
that employee similar types of personnel configurations. 
However r as a proposed program deviates more and more from con- 
ventional practice, the utility of conventional program data for 
guiding cost prediction diminishes. 

A category that represents a small proportion of a total 
budget deserves less attention since even relatively large errors 
in accounting for or estimating these costs will not significantly 
effect the total cost figure. The temptation to diligently 
obtain the latest cost figures for paper clips and pencils and 
settle for rough estimates of expensive computer time should 
always be resisted. 

The outline provided contains one general functional 
categorization scheme for Navy education and training R&D cost 
analysis. It is eclectic in that it includes components found in 
several but not all costing schemes. Many such schemes do not 
separate the replacement function from operation activities r but 
that decision can be made once data are obtained and levels of 
aggregation can be considered. 

C. Program Cost Analysis Summary 

Once cost data have been categorized, collected, and pro- 
cessed, one useful way to array or report the results is to 
construct a summary matrix. Such a matrix helps transform the 
data .into information by identifying recurring and non-recurring 
costs as well as fixed and variable costs. It also helps deci- 
sion makers to review and compare instructional alternatives on a 
functional cost or program-oriented basis. 

In Figure 4, general cost categories are listed on the 
horizontal axis. These are obviously gross categories, but this 
is intended to be used a summary or a display of aggregated cost 
data. The vertical axis displays the previously defined 
activities with a subtotal now added to emphasize and i^solate 
non-recurring dollar costs. 

The preceding sections have illustrated the all-important 
concern for initial emphasis on the function(s) and purpose(s) of 
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any cost-effectiveness analysis.. Cost categorization and matrix 
reporting schemes r such as the one shown in Figure 4, are impor- 
tant tools but should not be employed before the questions or 
problems are well defined. 
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